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Abstract 

Background: Announcements of interinn analyses of a clinical trial convey information about the results beyond 
the trial's Data Safety Monitoring Board (DSMB). The amount of information conveyed may be minimal, but the fact 
that none of the trial's stopping boundaries has been crossed implies that the experimental therapy is neither 
extremely effective nor hopeless. Predicting success of the ongoing trial is of interest to the trial's sponsor, the 
medical community, pharmaceutical companies, and investors. We determine the probability of trial success by 
quantif/ing only the publicly available information from interim analyses of an ongoing trial. We illustrate our 
method in the context of the National Surgical Adjuvant Breast and Bowel (NSABP) trial, C-08. 

Methods: We simulated trials based on the specifics of the NSABP C-08 protocol that were publicly available. We 
quantified the uncertainty around the treatment effect using prior weights for the various possibilities in light of other 
colon cancer studies and other studies of the investigational agent, bevacizumab. We considered alternative 
prior distributions. 

Results: Subsequent to the trial's third interim analysis, our predictive probabilities were: that the trial would eventually 
be successful, 48.0%; would stop for futility, 7.4%; and would continue to completion without statistical significance, 
44.5%. The actual trial continued to completion without statistical significance. 

Conclusions: Announcements of interim analyses provide information outside the DSMB's sphere of confidentiality. This 
information is potentially helpful to clinical trial prognosticators. 'Information leakage' from standard interim analyses such 
as in NSABP C-08 is conventionally viewed as acceptable even though it may be quite revealing. Whether leakage from 
more aggressive types of adaptations is acceptable should be assessed at the design stage. 

Keywords: Predictive probabilities, DSMB, Interim analyses. Operational bias, Prior distribution, Bevacizumab, Colon cancer 



Background 

Interim analyses of clinical trials convey information be- 
yond the sphere of confidentiality of a trials Data and 
Safety Monitoring Board (DSMB), A press release from 
the sponsor may announce, 'Good news! The DSMB rec- 
ommends that the trial continue/ or the opposite reac- 
tion: 'Shares of Oncothyreon dove over 40% on Tuesday 
after the [announcement] that a phase III clinical trial 
for Stimuvax [would] continue for several more months' 
[1]. Such public announcements indicate that the results 
have not crossed a stopping boundary. An implication is 
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that the experimental therapy is neither extremely ef- 
fective nor hopeless nor obviously unsafe. In the case of 
Stimuvax, Wall Street had convinced itself that the prod- 
uct was so good that crossing an early efficacy boundary 
was a foregone conclusion [2]. 

Trial sponsors are obviously interested in the results 
of interim analyses. Other interested parties include trial 
investigators, patients, venture capitalists, stockbrokers, 
and the sponsors competitors. When processing an- 
nouncements of DSMB recommendations, and if left to 
their own devices, people process information badly, 
usually overestimating or underestimating. In the words 
of Daniel Kahneman, 'intuitive impressions of the diag- 
nosticity of evidence are often exaggerated' [3]. We illus- 
trate a methodology for incorporating only the publicly 
available results of interim decisions in order to find the 
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probability the trial will be successful We assumed here 
that the interim analysis rules from the trials protocol 
are available to the assessor. 

Finding the probability of trial success based on interim 
information requires the probability of success separate 
from, or prior to, that information. The probability of trial 
success based on the prior and currently available infor- 
mation, and not assuming any particular hypothesis 
about treatment effects is called predictive power,' and 
contrasts with both traditional power and conditional 
power. Traditional power, assessed at the design stage, 
assumes a particular clinically significant' treatment ef- 
fect [4]. Conditional power, assessed at an interim point 
in the trial, conditions on the current data, and assumes 
a particular point estimate (usually the maximum likeli- 
hood estimate) for the treatment effect. Both traditional 
power and conditional power may be considered across 
a variety of scenarios, but the assumed treatment effect 
in each scenario is fixed. By contrast, predictive power 
is a weighted average of power over the current poster- 
ior' distribution of the unknown treatment effects [5]. 

The distribution of treatment effects prior to observing 
any information from the trial; that is, the prior distribu- 
tion, depends on phase 2 results; evidence from trials in- 
volving similar treatments, such as drugs in the same 
class; and evidence from trials with the same treatment 
but different diseases. It also depends on the person mak- 
ing the assessment, which gives the Bayesian approach an 
explicitly subjective flavor. One person may be pessimistic 
about successfully developing a treatment for a particular 
disease, whereas another may be optimistic. The hypothe- 
sized effect used in powering a clinical trial tends to be op- 
timistic; for example, it may be based on the results of 
phase 2 trials. Such an estimate is biased and subject to 
regression to the mean [6,7]. Evidence for this statement 
is that phase 3 clinical trials have at least 80% power, 
and yet no therapeutic area achieves a success rate even 
approaching this level [8]. 

An honestly assessed prior distribution allocates some 
probability to treatment effects that are less optimistic 
than the one used in powering the trial. It might also as- 
sign some probability to more optimistic treatment ef- 
fects. Perhaps the prior distribution is symmetric about 
its mean, and the mean is the effect at which the trial is 
powered. The excess in power above the mean does not 
begin to compensate for the loss in power below the 
mean. Therefore, properly incorporating uncertainty, the 
predictive calculations give a probability of success that 
is less than the calculated power. An empirically based 
rule of thumb is that trials built to have 80% power have 
a success probability of 60%. Moreover, the mean of the 
prior distribution is usually too optimistic. Discounting 
for excessive optimism decreases predictive power still 
more, to even less than 50%. 



We illustrate our methodology using the National Surgi- 
cal Adjuvant Breast and Bowel Project trial C-08 (NSABP 
C-08) [9]. This trial evaluated the benefit of adding bevaci- 
zumab (Avastin®; Genentech/Roche) to modified infiasional 
fluorouracil, leucovorin, and oxaliplatin (mFOLFOX6) in 
treating colon cancer. 

Methods 

NSABP C-08 study design 

NSABP C-08 randomized patients with stage II or III 
colon cancer equally to mFOLFOX6 (control group) ver- 
sus bevacizumab plus mFOLFOX6 (experimental group). 
The primary endpoint was disease-free survival (DFS) 
measured from randomization to the first recurrence, 
second primary cancer, or death from any cause. Based 
on previous studies, the control event rate was expected 
to be highest in the first 3 years. In total, 2,632 patients 
were planned to give 90% power to detect an anticipated 
25% reduction in the hazard of recurrence. The primary 
analysis would be a one-sided log-rank test. 

The trial's target monthly accrual rate was expected to 
increase from 63 to 105 patients over the first 2 years, 
completing accrual within 30 months. The protocol- 
defined final analysis was to occur at 592 DFS events. 
The first interim analysis was planned after 148 DFS 
events (25% information), approximately 2 years after 
trial initiation. Subsequent analyses were planned every 
6 months. Neither the number of analyses nor the sig- 
nificance level of the final analysis were predetermined, 
but would depend on alpha spending [10]. However, the 
investigators anticipated six interim analyses, which is 
what we assumed in our simulations. The stopping rule 
used asymmetric boundaries for superiority and futility 
indexed by one-sided P- values (Table 1). Maximal trial 
duration was expected to be 5 years, including 2.5 years 
of follow-up after the final patient was accrued. 

Statistical simulations 

We simulated trial results based on the publicly available 
specifics of the NSABP C-08 design and the results of 
previous interim analyses. We assumed particular treatment 
effects (see below) to generate virtual patients based on the 
assumed accrual rates, and followed their DFS events over 
time. These virtual patients experienced DFS events over 
time, depending on the assumed effect of their assigned 
treatment. Each simulated trial stopped or not at interim 
analyses, according to the boundary defined in Table 1. If it 
did not stop early then we compared the log-rank P-value 
to the final one-sided significance level of 0.0246, as shown 
in Table 1. When a simulated trial stopped at an interim 
analysis we recorded that fact and the result. For each 
simulation, we recorded whether, when, and why the trial 
stopped, and the eventual statistical conclusion. We iter- 
ated this process 70,000 times for each assumption about 
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Table 1 National Surgical Adjuvant Breast and Bowel Project trial C-08 (NSABP C-08) interim monitoring bounds in 
terms of one-sided P-values 



Interim analysis 


Estimated time, years 


Stop for futility if 
right-sided P-value < 


Stop for superiority if 
left-sided P-value < 


Number of events 


Estimated accrual 
at interim 


1 


2.0 


0.05 


0.00025 


148 


2,006 


2 


2.5 


0.25 


0.0005 


-220 


2,632 


3 


3.0 


0.5 


0.001 


-312 


2,632 


4 


3.5 


0.5 


0.001 


-398 


2,632 


5 


4.0 


0.5 


0.001 


-473 


2,632 


6 


4.5 


0.5 


0.001 


-538 


2,632 


Final 


5.0 


N/A 


-0.0246 


592 


2,632 



treatment effects. The proportion of simulated trials with 
outcomes of interest (such as that the trial continues at 
the third interim analysis, but stops for superiority of the 
experimental arm at the fourth interim analysis) are the 
estimated probabilities of those outcomes. 

We followed the assumption in the trials protocol 
when generating the time that each virtual patient would 
experience a DFS event. Specifically, we assumed a piece- 
wise exponential distribution with control hazard rate 
0.089 for the first 3 years, reducing to 0.039 in subsequent 
years. When generating patients in the bevacizumab group, 
we assumed a constant hazard ratio versus control. 

The treatment effect was the reduction in hazard due to 
bevacizumab. Any value from negative infinity to 100% 
was possible, but the practical range was 0 to 40%. We 
assigned (prior) weights to values in this range, in incre- 
ments of 5%. Figure 1 gives example relative weights. 

The custom' prior distribution in Figure 1 was that 
assessed (by author DAB) during the NSABP C-08 trial. 
This prior was not based on any knowledge of trial re- 
sults, but considered publicly available information from 
other colon cancer trials and from other trials of bevacizu- 
mab [9,11-15]. The available information was consistent 
with reductions of 10% and 15%, with little suggestion that 
the reduction could be 30% or greater. 

The other prior distributions shown in Figure 1 were 
used to address the sensitivity of our conclusions to the 
prior assessments. 'Simple' gave equal weight to each pos- 
sible reduction; pessimistic' concentrated weight on little 
or no reduction; 'moderate' focused on 15% and 20% re- 
ductions; and optimistic' gave most weight to substantial 
reductions. 

Results 

Prior predicted power 

Figure 2(A) shows the cumulative predictive probability 
of success (bottom five lines) and the cumulative pre- 
dictive probability of futility (top five lines). The steep 
increase in the cumulative probability of success at the 
final analysis reflects the larger P-value boundary used at 
the end of the trial compared with those used for interim 



analyses. Based on the custom prior distribution, the over- 
all probability of statistical success was 47.5%, much less 
than the trial's advertised power of 90%. The predictive 
probability of success is substantially lower than the power 
because it incorporates uncertainty in the hazard reduc- 
tion, and gives weight to hazard reductions of less than 
25% (at which power was evaluated). The cumulative 
probability of success by the final analysis based on the 
optimistic' prior was 76%, which was still less than the tri- 
al's planned power. This is despite the fact that this prior 
distribution was centered close to the 25% reduction as- 
sumed in the protocol. The decrement in the probability 
of success is due to the uncertainty that is explicitly con- 
sidered in the prior distribution. 

The increases in the cumulative probability of futility 
from the first and third interim analyses reflect the less 
stringent futility stopping bounds subsequent to the sec- 
ond interim analysis; futility is a possibility only at interim 
analyses. Based on the custom prior, the overall probabil- 
ity of futility was 20.4%. This ranged from 5.8% with the 
optimistic prior, to 34.3% with the pessimistic prior. The 
cumulative custom probability the trial would remain un- 
decided at the final analysis was 32.1%. The cumulative 
probability of undecided' ranged from 18% for the opti- 
mistic prior, to 37.4% for the pessimistic prior. 

Figure 2(B) shows the predictive probability of success 
and the predictive probability of futility, conditional on 
the previous interim analysis decision being to continue. 
Figure 2(C) shows the predictive probabilities of even- 
tual success and the predictive probabilities of eventual 
futility. Figure 2(C) indicates that the probability of eventual 
success in the trial, given that it continues, decreases only 
slightly over time. In this sense, announcing the DSMB's 
recommendation to continue is not very informative. 

Actual trial conduct 

Figure 3 shows the timeline of NSABP C-08 and our 
predictions. The trial began in September 2004 [9]. In 
February 2006, accrual was temporarily halted to allow 
the DSMB to review adverse event data. Accrual resumed 
in May 2006. Accrual was faster than expected, and was 
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Treatment Effect 

Figure 1 Prior distributions of the treatment benefits given in terms of relative strengths of belief in hazard reduction due to 
bevacizumab. Abbreviations s, simple; p, pessimistic; o, optimistic; m, moderate; and c, custom. 



completed in October 2006 with 2,710 patients enrolled 
[9]. The first interim analysis was conducted in April 2007. 

Our analysis of the predictive probability of the trials 
success was conducted in June 2008, after the third in- 
terim analysis. We made our software and custom' prior 
distribution public on June 12, 2008 via a webinar spon- 
sored by UBS Bank. Our predictive probabilities for the 
fourth interim analysis were 9.8% for success and 3.8% 
for futility. At that time, our predictive probabilities of 
eventual success, stopping early for futility, and continu- 
ing to completion without statistical significance were 
48.0%, 7.4%, and 44.5%, respectively. 

Our analysis depended on the assumed prior probabil- 
ities, the roots of which are subjective [16]. An Excel 
spreadsheet that carries out the calculations described in 
this article is available at http://www.berryconsultants.com/ 
wp-content/uploads/2014/02/Unknown-Hazard-Reduction. 
xls (Figure 4). It allows the user to input a prior distribution 
by entering a weight for each of the indicated hazard reduc- 
tions. The sum of the weights is arbitrary; only their relative 
sizes matter. The weights convey confidence in the corre- 
sponding hazard reductions, and should reflect the available 
evidence. For example, entering 10' for the 25% reduction 
and V for the 0% reduction means that the prior belief in 
the former is twice as strong as for the latter. Once the 
prior distribution is entered, the probabilities of success 
and ftitility and other conditional measures are calculated 
and displayed. Varying the prior weights shows how the 



probabilities of the various outcomes depend on these 
weights. For example, entering zero weight in all but a 25% 
reduction gives the overall power of the trial evaluated at 
this clinically important difference', which is 92%. 

Owing to more rapid accrual than expected, the actual 
NSABP C-08 had only four rather than the planned six 
interim analyses [9]. The trial continued to completion, 
and the final analysis, conducted in April 2009, showed 
that the bevacizumab arm reduced the hazard for DFS 
events by 11%. This was not statistically significantly dif- 
ferent from control (hazard ratio 0.89, 95% confidence 
interval 0.76 to 1.04) P- value = 0.15) [9]. This resuft was 
consistent with our custom prior, which gave the most 
weight to a 10 to 15% reduction in hazard. 

Discussion 

Predicting whether an ongoing clinical trial will be suc- 
cessful is of interest to the trial sponsor, the scientific 
community, other pharmaceutical companies, and inves- 
tors. When the NSABP C-08 trial was initiated, the owner 
of bevacizumab was Genentech. As the trial proceeded, 
Hoffman-LaRoche was in negotiations to purchase the 
50% portion of Genentech stock that it did not already 
own. Hoffman-LaRoche became aware of our software as 
shown in Figure 4, and they contracted with us to update 
the software using the actual accrual from the trial (data 
not shown, and not publicly available at the time) rather 
than using accrual projected from the protocol. 
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1 2 3 4 5 6 Final 
Interim Analysis 



Figure 2 The symbols use in all panels refer to the prior weights indicated in Figure 1/ Abbreviations s, simple; p, pessimistic; o, 
optimistic; m, moderate; and q, custom. The top five lines and right y-axis show the futility results. The bottom five lines and the lefty-axis 
show the success results. (A) Predictive probability of success or futility at or before the indicated interim analysis. (B) Probability of success or 
futility at the indicated interim analysis conditionally on 'Continue' at the previous interim analysis. (C) Predictive probability of eventual success 
or futility assuming 'Continue' at the previous interim analysis. 

V ) 



Hoffman-LaRoche had access to our software after the 
fourth interim analysis. Using their prior probabilities, 
they concluded a 55% probability of trial success and 
used that prediction to set the price they offered for the 
Genentech stock [17]. Genentech determined the prob- 
ability of trial success to be 61% [17]. We do not know 
the basis of this assessment, but it is close to that of the 
moderate prior defined in Figure 1, and it too is substan- 
tially less than the trials power of 90% claimed in the 
protocol (which assumed a 25% reduction in hazard). 

We used the NSABP COS trial as an example to illus- 
trate a general approach. Not stopping at an interim 



analysis provides some information about the trials pri- 
mary endpoint and its eventual success. The amount of 
information depends on the trials stopping boundaries, 
and is usually small. We demonstrated how the informa- 
tion gained from the announcement of an interim ana- 
lysis that the trial is continuing can be used to make and 
update predictions about the final trial conclusion. 

NSABP COS had more planned and actual interim ana- 
lyses than is typical in phase III trials. A greater number of 
interim analyses provides more opportunity for updating, 
but not necessarily more overall interim information. The 
number of opportunities for updating may not be the 



Sept 04 Feb 06 Mar 06 Oct 06 Apr 07 Oct 07* Apr08*Jun08 Oct 08* Apr 09 

Figure 3 Timeline of actual National Surgical Adjuvant Breast and Bowel Project trial C-08 (NSABP C-08) trial. Asterisk indicates 
approximate timing. 
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Nr. 


Description 


0.00 


0.05 
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1 1 1 
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1 1 




2 
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4 


2 


1 


1 


1 0 
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3 
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1 


1 
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3 


3 


2 


1 


1 0 


6 


4 


Optimistic 


0 


1 


2 


3 


4 


5 


5 


2 1 
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2 
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5 


3 


4' 


5' 


5 


4 


3 


2 


1 


0 


11 
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0.111 


0.148 
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0.185 


0.148 


0.111 
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0.037 


0.000 
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eventual 


eventual 


eventual 


14 
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sucjundcc 


futjundec 


fail|undec 


IS 








0.023 


0.012 


0.965 


















2 


0.074 


0.070 


0.856 


0.056 


0.059 


0.474 


0.198 


0.328 


17 






3 


0.150 


0.158 


0.693 


0.101 


0.105 


0.477 


0.160 


0.363 


la 






4 


0.204 


0.181 


0.615 


0.098 


0.038 


0.480 


0.074 


0.445 


It 






5 


0.242 


0.195 


0.562 


0.089 


0.025 


0.469 


0.041 


0.490 








6 


0.269 


0.204 


0.527 


0.071 


0.017 


0.453 


0.017 


0.529 


2i 






7 


0.475 


0.204 


0.321 


0.438 




0.438 




0.562 



Figure 4 Screenshot of Excel spreadsheet demonstrating the calculations described in the text. Entry '5' at cell lOB refers to the entry in 
cell 7A, and sets the prior probabilities to those labeled 'Custom' in row 7. These were the probabilities of the author (DAB) during the early 
conduct of National Surgical Adjuvant Breast and Bowel Project trial C-08 (NSABP C-08), and were based on the publicly available information 
about bevacizumab and its potential efficacy in treating adjuvant colon cancer, as described in the text. 'Interim analysis 7' is actually the final 
analysis. The emboldened type in row 18 indicates the probabilities of eventual success, futility, and failure after the fourth interim analysis. 
This spreadsheet is available at http://www.berryconsultants.com/wp-content/uploads/2014/02/Unknown-Hazard-Reduction.xls. It allows 
users to input their own probabilities, and to experiment with other prior probabilities to see how they affect conclusions. The column 
headings are 'sue' (the probability of success at the current row's interim analysis); 'fut' (the probability of futility at the current row's interim 
analysis); 'undec' (the probability of making no decision at the current row's interim analysis and continuing the study); 'suc|undec (the 
probability the trial will be declared successful at that row's interim analysis, given no decision at any previous analysis); 'fut|undec (the 
probability the trial will be declared futile at that row's interim analysis, given no decision at any previous interim analysis); 'eventual suc| 
undec (the probability the trial will be declared successful at the current, future, or final analysis, given no decision at any previous analysis); 
'eventual fut|undec (the probability the trial will be declared futile at the current, future, or final analysis, given no decision at any previous 
analysis); and 'eventual fail|undec (the probability the trial will fail to show statistical significance given no decision at any previous analysis). 



same in other phase III trials, but the approach and the 
methodology is the same. Moreover, as regards efficacy, 
and assuming that the total amount of type I error allo- 
cated to interim analyses is similar, the total amount of in- 
formation gleaned from interim analyses is also similar. 
The difference for another trial is that the information 
may be revealed at a different rate over time. 

Trial designs vary more in their approach toward futil- 
ity stopping than they do in their approach toward effi- 
cacy stopping. The information from an interim analysis 
for futility is positive in the sense that not stopping 
bodes better for trial success, although the amount of in- 
formation may range from minimal to substantial. \Xn:ien 
stopping for either efficacy or futility are both possibil- 
ities at the same analysis epoch, as in the case of NSABP 
COS, then the information that the trial is continuing 
could be either positive or negative in relation to ultim- 
ate success. 

If a trial has no interim analyses, then no updating is 
possible. However, even then it is possible to calculate 
the predictive probability of trial success using prior in- 
formation about treatment effect. 

As our example NSABP COS suggests, the assessment of 
prior probabilities of the effect of experimental therapy is 



crucial, and prior information is usually more important 
than information that accrues from interim analyses. 
Assessing prior probabilities is inherently subjective, as we 
demonstrated by providing a variety of prior distributions. 
In practice, such variety will usually exist, and there may 
be even a wider range. In our experience, the worst asses- 
sors of prior probabilities tend to be those closest to the 
development of the experimental therapy; sometimes they 
have been entrapped' [6]. In our custom' prior we consid- 
ered the performance of bevacizumab in other trials of 
colorectal cancer, trials of bevacizumab in other cancers, 
trials in cancers involving similar agents, and the sensi- 
tivity of colorectal cancer to therapies more generally. 
This last point is crucial. It plays the dominant role in 
predictions involving diseases such as pancreatic cancer, 
glioblastoma multiforme, stroke, sepsis, and Alzheimer's, 
where the overwhelming proportion of phase III trials 
have failed. 

Having software that quantifies the information con- 
tained in announcements of interim analyses raises con- 
cerns about ethics and the need to preserve clinical trial 
integrity. Our approach does not provide more information, 
but rather indicates how to interpret the available informa- 
tion. In our experience, interpretations of announcements 
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of interim analyses are frequently wrong, and are usually 
overinterpretations, sometimes in the wrong direction. 
This applies to the trials' investigators and other experts 
in the field as well as to venture capitalists and Wall Street 
bankers. Having software that provides a more accurate 
and usually more tempered interpretation may have a 
beneficial and moderating effect. 

Although our main focus is on outside observers, soft- 
ware that accurately assesses the extent of 'leakage of infor- 
mation' in a clinical trials design may be an aid to clinical 
trial designers and trial review bodies. Our software could 
help them assess the extent and impact of information 
leakage on the trial. No leakage is good, but some leakage 
is necessary. The benefits of interim analyses to patients 
and to society generally usually greatly outweigh the nega- 
tive aspects of information leakage, but assessing the extent 
of leakage may lead to a modification of the design that 
lessens its potential impact on the trial s conduct. 

In a related vein, modern trialists are introducing adap- 
tations that are more complicated than group-sequential 
methods [18,19]. If these adaptations become public dur- 
ing the trial, and if the trial design is similarly known, then 
the observed changes in the trial may convey information 
about the relative performance of the treatment arms 
[20-22]. Trialists should assess the impact of such know- 
ledge outside the sphere of confidentiality of the trials 
DSMB, and may want to take steps to keep adaptations 
confidential or perhaps modify the design. 

Conclusions 

Announcements of interim analyses for a group-sequential 
trial design such as NSABP C-08 reveal that the interim 
data lie within a particular range. This information is poten- 
tially helpftil to clinical trial prognosticators and to clinical 
trial designers. Interim analyses such as those in NSABP 
C-08 are typically viewed as acceptable with regard to the 
amount of information conveyed [18,22]. Not all prognos- 
tication is done well, but any prognostication has the po- 
tential to influence the course of the trial. Our formal 
procedure based on Bayesian updating uses information 
that becomes available over time, in order to assess accur- 
ately the probability that a clinical trial will be positive. 
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